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ABSTRACT 



For 3 years, the Diagnostic and Evaluative Procedures course 
of an education college was offered by electronic mail as well as by a more 
traditional "chalk and talk" approach, enabling a comparison of the two 
approaches. In the electronic setting, papers were submitted and questions 
asked and answered electronically. In the traditional class, students were 
expected to attend and listen to prepared lectures or participate in class 
discussions or activities. For this evaluation , there were 78 participants 
in the electronic classes and 65 in the traditional classes. Both groups were 
pretested and posttested to assess knowledge of course content. No 
significant differences in knowledge gain were found between the classes, 
suggesting that it is possible to conduct the class electronically without 
loss of academic performance. Attachments include the description of the 
course, an evaluation survey, and analysis of co-variance results. (Contains 
26 references.) (Author/SLD) 
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Abstract 



For three years the Diagnostic and Evaluative Procedures course was offered by electronic mail as 
well as by a more traditional ”ch^ and talk" approach, enabling a comparison of the two ^proaches. Fn 
the electronic setting, papers were submitted, and questions asked and answered electronically. In the 
traditional class, students were expected to attend and listen to prepared lectures or participate in class 
discus»ons or activities. Both groups were prete^ed and posttested to assess knowledge of course 
cont^. No £»gnificant dififerraces in knowledge gain were found between the classes, suggesting that it 
is possible to conduct the class eiectronicdly without loss of academic performance. 




Evaluation by Electronic Mail 



Internet usage is growing rapidly, estimated to be in the tens of millions of users (Miller, 1996). 
It should come as no surprise thoi, that businesspeople as well as educators have join^ in of^ng 
electronic courses (Buettner and de Moll, 1996; Hatley, 1996; Kelly, 1996; Nantz and Drexel, 1995; 
Warschauer, 1995). This use of electronic connections for academic pursuits has opened a new era in 
education, providing the opportunity for teaches to work with students both near and far as well as at 
many levels, ranging from elmentary school (Martinelli-Zaun, 1993; Partridge, 1995; Weaver, 1995) to 
high school (Barr, 1994; Partridge, 1995) and college (Wilson and Mardi, 1995). 



Advantages 

Garride (1996) noted the impact of dectronic mail (e-mail) on the improvement of communication 
skills, as well as convenience and co^ effectiveness. Barr (1994), Gabriel (1983), and Pitt (1996) wrote 
of the opportunity to interact with students individually. The casual nature of e-mail (Kinkead, 1987) 
breaks down bairim among dasa'ooms, students, and tubers (Brienne and Goldman, 1990). Although 
Mclnt 5 fre and Tlusty (1995) found no direct evid^ice that e-mail provided an effective measure of 
reflection, Brienne and Goldman (1990) rqmrted that electronic mail can contribute to students' critical 
thinking. As students analyze their data and communicate with others, they are exposed to new opinions 
and have to think critically about their own and others' analyses. Deal (1995) pointed to students' 
syntherized learning and increased self-assessment skills, as well as a better understanding of the students' 
concerns, by the tocher. Electronic mail has also changed some students' perceptions of themselves. 
Newman (1989) reported that h^ students' self-esteem improved because of the interest oth^ students 
showed in their woik. 



Disadvantages 

Not all electronic mail experiences are positive. For example, Newman (1989, p.792) wrote that 
students sometimes found it difficult to write to strangers. Also, some found it difficult to set aside time 
to use university computes. Sometimes it is difficult for the teacher as wdl as for students (Weaver, 
1995). As one teach^ stated, she fek like she was in over hw head (Newman, 1989). Setting up the 
equipment was fiustrating, particularly when technical support was unavailable. Even when it was 
available, there still could be limited acce^ and unrdiabile equipment (Garside, 1996). 

Learning to use the software can also be disconcerting, sometimes requiring extensive training and 
support (Fine, 1991). Honq? and McMillan (1993) noted that learning to use ffie network requires an 
int^rive iiwestmart in time, v^ch few teachers have available. Harris (1993) r^rorted that students also 
spaid a large amount of time, sometimes 6-10 hours per week working on-line to complete assignments. 

Tedinolc^ also require a signified finaiK^ investment (Fine, 1991). Gabrid (1983) found that 
it could also be diffioilt to measure students' gains from using dectronic technology. In addition, he and 
Garside (1996) noted teacher and user resistance to the use of technology since not everyone wants to 
work electronically. Finally, Barr (1994) wrote that critics have argued that the focus of electronic mail 
projects tmds to be on the process rath^ than the contort. As a consequence, many questions and answers 
may tend to be aipaficial. 
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The Electronic Class 



Despite the possible drawbacks, flexibility for students was an important motivation for offering 
the diagnostic and evaluative procedures course by electronic mail. The students, nearly all senior 
el^entary education m^ors who wae in thar prq}rofes»onal teaching semester, spent mornings teaching 
off campus before returning to the university for class. The physical and psychological drain of teaching 
unda* »ipavis(Hi for a halfday followed by a return trip to campus for a two-hour class pro\aded a good 
opportunity for a non-campus approach. 

On an expaimattal 1^^, tte dectronic course was first ofifaed in the Fall of 1994 to 1 3 volunteers 
out of 49 studatts who had enroll^ in the two class sections of the course. The other 36 students opted 
for the more traditional "talk and dralk" format with multiple dioice tests, taught during the same time slot 
but 1^ a diffaait instructor. The course has since been r^ularly offered in the Spring and Fall semestas, 
with the Spring, 1 996, semester bang the most recently complaed. There were 78 participants in the 
dectromc classes and 65 in the traditional classes. Most of the studaits in both classes were young white 
females in thar satior year of the elemaitary education program. 

The abilities of the e-mail studaits raided fipom "newbies" to computa literate, and fi-om computer 
non-ownas to ownas. The neophytes necessarily required more technical support than the others, but 
once communication was established, the students' confidence grew quickly. The students who did not 
own or have easy access to computers were able to use those in university PC or Macintosh labs, making 
it possible for them to use eitha platform to communicate. 

Electronic course requiranents (See syllabus following the refaences.) included reading the 10- 
diapta text (Gronhmd, 1993) and writing responses to each of the chapta objectives. Although students 
could type their refuses directly during real time, most preferred to type them into a word procesi^g 
document, save them to disk, and then send the files when convenient. Students were able to post 
messages, questions, assignments, and other communications at any time although th^ wae not limited 
exdusively to e-mail; they wae allowed to call the instruaor at home or at work, or to visit when on 
campus. Few i^daits availed thaiselves of these opportunities, prefaring instead to work at a distance. 
The electronic mailbox was checked regularly so that students would receive prompt replies. 

Another requiranent of the class, which in lata semesters was made an option, was a research 
papa on an assesanait topic of the ^dait's dioice. After recdving approved for the topic, students could 
vi9t the library, conduct tlffi res^di, and then aibmit the document ova e-mail. One project 0ata two) 
was also required. Sevaalc^ons wae provided, but most ofthe students chose to prepare tests. Otha 
students chose to survey practicing teachas u»ng an instrument provided by the instructor. While the 
tests, checklists, or scales wae salt ova e-m^, the complaed survey forms wae simply dropped off on 
campus to save typi^ than into a word procesang document. Tests wae taken in a computa lab in the 
education building to maintain seoirity. Students were given up to five attempts on a multiple-choice test 
of 20 questions, each sa bang ramiomly drawn fiom a huge po(^ The highest score obtained was counted 
as the studaif s final exam grade. 



Findings and Condusion 



To daamine whaha tl» studaits in the electromc cla^ wae paformbig comparably with studaits 
in the traditional course, preteas and postteas wae given, smila in format to the regula teas 
adminiaaed to both groups for thar course grades. An analyss of covariance (ANCOVA) was run on 
tte sones usi^ the poat^ scenes as the response variable and die praea scores as the covariate. Since 
the ANCX)VA technique involves features of both the analy»s of variance and r^ession, {^sumptions for 
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both were tested using the NCSS statistical program, vmion 6.0.21 (Hinto, 1996). The assumption of 
random selection is not practical for most courses, but the two groups were comparable for all practical 
puiposes. N(Mmali1y and homoscedastidty across all groups was verified using the Omnibus Normality of 
Residuals and Modified-Levene Equal- Variance tests. Homogeneity of regression was observed in a 
scatterplot of both pretests and posttests and thdr trend lines. Therefore, the asaimptions required for 
ANCOVA seemed to be reason^ly well met. 

The test indicated that the null hypothesis of no statistically significant difier^ce between the 
traditional (ac^usted mean of 13.0, n=65) and dectromc (ac^usted mean of 13.5, n=78) classes' scores could 
not be rgected at the 0.05 level (pf=0.24). It is concluded, then, that ofiering the course throu gh electronic 
mail did not appear to hinder the performance of the students, to the extent measured by the multiple- 
choice tests, suggesting that the electronic course offering provided a flexible alternative for learning, 
poshly with some nona«:adamc advantages over the tradition^ ^proach. The posttest performance was 
comparable to the traditional offering, suggesting that the electronic approadi is not only viable, but may 
ev«i be preferable. 
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Course Prefix and Number 

Course Title 

Credit 

Semester and Year 
Instructor 

Office Location 
Office Hours 
Telephone 



College of Education 
Departmoit of Educational Leadership 
(revised 1/13/96) 

EDFN4205 

Diagnostic and evaluative procedures in education 
2 hours 
Spring, 1996 

Rob Kennedy, Ph D., Associate Professor of Educational 
Foundations 

Larson 204B 

By appointment 

569-xxxx (UALR), 327-xxxx (home), RLKENNEDY@xxxx.xxx 
(e-mail) 



Course Description 

A study of fundamental statistical concepts and thdr use in understaiKling standardized test re^ts. 
Emphasis on the exploration of qualitative methods and evaluating and reporting progress. 

Course Objectives 



Survey practicing public el«mntary school teachers for their views on important issues and contait in 
assessment. 

Plan and con^ruct test^quesdons appropriate for the grade levels for which you will be certified. 
Calculate grades and other measurements. 

Investigate curr«it and anticipated trends in assessment, including pafonnance-based assessment 
involving portfolios ami paformance events. 

TextSj Readings^ and Instructional Resource 

Required Text 

Gronhind, N. E. (1993). How to Make Achievwnent Tests and Assessments (5th ed. V Boston. Allyn 
and Bacon. 
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Asagnments. Evaluation Procedures, and Grading Policy 
Course Requirements 

Students who demonstrate a commitmait to the course through attendance, participation, riding, 
studying, and otherwise applying themselves to the course will benefit in direct proportion to that effort. 

In other words, "You get out of it what you put into it." 

Note that attaidance is required the first two dass meetings and the last class meeting (April 29th). Your 
presence is needed and wiU be counted as part of your grade, so please make every effort to attend the 
last day. 

Evaluation Techniques/Concepts used for Grading 

Final examination (50%) 

Homework (10%) 

Projects (2 @ 10% ^di) 

Regular communication (10%) 

Attendance and completion of in-class exercises April 29th (10%) 

Final examination (50%1 

The final examination will be ovm' the entire course. The exam is a computer test which will be taken in 
the computer lab in Larson Hall. You will be able to take the test a maximum of five times, with your 
highest score counted. You may take the test less than five times, but your highest score will still be the 
one counted. The test will be multiple-choice, dosed book, and fi-om material in the text. You may take 
the test whenever you are ready, but remember that it is comprehenave, so you may wish to wait until 
you have studied all of the content before uang more than one of your five tries. You will be able to 
continue your trials through Monday, May 3, 1 995. Note: Based on the experiences of num^ous 
students, spreading out the testing after you are ready is generally more effective than taking multiple mns 
of the test at one time. Do NOT try to take all five runs in one day. This has not been a atccessful 
strategy. 

Homework (10%) 

You will be expected to complete the objective spedfied in each of the 10 chapters in the text for the 
course. As you complete each chq>ter, you \dll submit your responses electronically. You will recdve 
full credit on each assignment unless I note that you should make corrections or redo it. Even in those 
circumstances, you will still recdve full a*edit wh^ acceptable revidons are made. 

Projects (20%) 

You will need to select and complete two prefects from the list below. Your completed projects will be 
submitted dectronically. 

1 ) Review examples of tests, checklists, scales, and/or portfolios. Then devdop dther a typed test, 
checklist, scale, or portfolio, induding objectives, depending on your interest. 

O 

ERIC 




2) Survey 5 practicing elemratary school teach^ for their views on important issues and cont^ in 
assessment. The survey instrument is induded in this outline. Responses to the surv^s may be 
handwritten. You do NOT need to atbmit the airveys electronically. 

3) A research paper is included as an option for a number of reasons. A research paper provides you 
with the opportunity to pursue an area of your into'est (relative to the class topics) which otherwise 
may receive less attration than you dem warranted, due to the time constraints of the course. In 
addition, research skiUs will end)Ie you to continue studying indq)endently as a life-long leamo-. 

Also, it is worthwhile to demonstrate your ability to dft through the vast array of publidied lit«'ature, 
to ferret out the gems of wisdom, and be able to communicate the findings to others. The paper 
needs to be a minimum of 2-3 typed pages in length, including a list of at least five references. AP A 
and MLA styles are acceptable as well as any other consists and neatly ]n%sented format. The 
research paper may also be submitted electronically. 

Some r^eardi paper ideas oth^ stud^ts have researched; 

Portfolio Asses^nrat 
Bias in Testing 

Assessment for the Hispanic ESL Learner 

Norm-Referenced Testing vs. Criterion-Referenced Testing; Rank vs. Skill 
Test Anxiety; What Can be Done About It? 

An Overview of Assessment in Adult Education 
Issues in Formative Assessm^ 

Achievement Tests and Alt^atives 

Asses»nent and Evaluation of the Afifective Domain in Education 
A Brief Review of the Rdiability and Validity of the Myers-Briggs Type Indicator 
Att^tion Defidt Disorder and Attention Deficit Disorder with Hyperactivity and its Role in the 
Classroom 

Alternatives in Language Arts Assessment 

In Search of Meaningful Reading Assessments 

Performance Assesanent; Completing the Profici«icy Picture 

Literacy Portfolios; What Exactly are They? How ^Qfective Will They be for Asses»nent? 

Reading Difficulties; Diagnods and Instruction 
Standardized Testing; Helpful? ... Or Harmful? 

Reading Disabilities and Providing Instruction 

Assessment in the Literacy-Based Classroom 

Diagnodng Reading Difficulties 

Whole Language Assessment 

Signs and Symptoms of Attention Deficit Disord^ 

AHemative As^s»nents 

Performance Assessments; An Educational Fad or Here to Stay? 

4) Create a page similar to one of the last sevo'al pages of this syllabus in which you explain to foture 
students how to use a word procesang prc^ram or a differait int^ace between your computer and 
the internet. Be sure to have someone else try out your set of steps to be sure that they work 
correctly. If you discova* any orors in the checklists at the eid of this document, please let me know 
so they can be corrected. 



Some checldist ideas oth^ students have devdoped: 



Saving tests for e-mdl using WP 

How to use MS-DOS Shdl Word Processing 

Using the Apple lie Network 

Word Procesi^g with AppleWorks 

Checklist on rduming an e-mail message 

AOL installation 

U»ng Spell-Check on WP6 

To Use WordPerfect 2.0 or 2. 1 on the MAC 

How to access e-mail from the VAX 

To use WP6/Win 95 on IBM 

Checklist for sending and receiving files 

To use Microsoft Word on the MAC 

Re gular communication f lO%I 

To otcourage you to r^ularly participate in class activities, you will be e;q)ected to submit some work 
weekly, excq>t in emergences. The rationale for this is that, unfortunately, some students tend to 
procrastinate on thdr asagnments and eventually find themselves camming in a lot of work in a diort 
time, to the detriment of thdr learning. I want to hdp you learn, so I am reqturing that you make this 
effmt. Note that turning in a chaptc asdgnmot or othc work every week is a minimum; it is not 
necessarily sufficicit for you to complete the course on sdiedule. You may need to submit more work 
some weeks in order to fin ish on time. 

Additionally, it is important that you regularly read your electronic mail. Although points will not be 
assigned for this responribility, your electronic mail vrill likdy be the most common means for 
communication between you and me. I suggest that you check in at least once a wedc to see if you have 
any mail. You will know immediatdy after logging in whether you have any new mail or not, so it will 
require only a small amount of time to find out. You will also fed more a part of the dass, particulariy if 
you leave mail r^plarly as well. 

Attcidance and compldion of in-dass exercises April 29th (10%) 

There are a few follow-up evaluative exerdses which you will need to be available for at the end of the 
course. It is very important that you contribute to the course in this way. These functions are important 
enough for evaluation and research purposes to assign them 10% of the credit to be awarded in the class. 
Please plan to partidpate. 

Grading Scale 

90-100 A 

80-89 B 

70-79 C 

60-69 D 

Below 60 F 
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Xni. Class Policies 

Any of the assignments may be returned to you to be revised or redone if they are deemed unacceptable 
by the instructor. The normal reason that papers are r^umed for being unacceptable is the student's lack 
of following directions. The directions given are minimized to allow you the freedom to be creative and 
to use and develop critical thinking skills, but the directions which ARE provided should be followed. 

The purpose of the asagnm^s is to ^courage growth in your thinking and understanding of the area of 
assessmait, which is rapidly changing as accountability becomes more and more of an issue. An example 
of inadequate work concerns test construction. Some i^drats have listed only a s^es of questions as a 
test. While a stud^t who had not had this course might think that a test conasts only of questions, those 
questions out of context have little or no meaning. For the purposes of this class, it will be necessary to 
provide an assessment environment for your test, including a list of your objectives. An assessment of 
your test items will be much more accurate if their purposes are clear. 

XI\L Class Schedule 

The dates suggested hoe are to help you in complying your assignm^ts in a timely foshion. It will be to 
your advantage (as well as being required) to REGULARLY submit assignm^ts. Although some 
students have unfortunately demonstrated that the course acti\aties can be crammed into a short time 
frame, test results have suggested that learning of course information is damaged in the process. 



January 15 


Introduction, overview of course 


January 22 


Computer Lab presentation 


January 29 


Achievonent testing and instruction 


February 5 


Planning the test 


Fdrruary 12 


Writing selection items; multiple choice 


Fdrruary 19 


Writing selection hems; true-false 


February 26 


Writing selection hems; matching 


March 4 


Writing supply items; short-answer 


March 1 1 


Writing supply hms; essay 


March 18 


Assembling, administering, and evaluating the test 


March 25 


Making p^ormance assessments 


April 1 


Interpreting test results 


April 8 


As»gning grades 


April 15 


Validity 


April 22 


Rdiability 


April 29 


Concluding exercises—Attendance required! 


May 3 


Official final exam date (1 ;30-3;30 pm) 

ALL WORK AND TESTS MUST BE COMPLETED AND SUBMITTED BY TfflS 
DATE. 
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Student 

EDFN 4205 Teacher Surv^ 

Diagnostic and Evaluative Procedures in Education 

I have be^ asked by one of my instructors at UALR to survey practidng public dementary school teachers for 
thdr opinions, from the practitioners standpoint, about the content which should be taught in the Diagnostic and 
Evaluative Procedures in Education course. Would you be willing to partidpate in this survey, if you have not 
already? Your name will not be used [and should not be writtei on this form], (If the response is no, you will 
need to find another teacher. If the response is yes, please write down the district employing the teacher and the 
grade levd(s) taught.) 



District: Grade Levd(s); 

Would you please respond to the following questions to the best of your knowledge and experience. We are 
trying to gatho* information to make the assessment class as realistic and meaningful as posdble. If you have any 
adtfitional comments you widi to make, fed free to add thm at any time. Ijst any comments to the right or on 
the back, indicating to which item the comm»)ts belong. Be sure that you can explain what each of these itms is 
(See Tuckman if you are unsure.). 

1 . Do you write behavioral objectives as part of your planning for tests? 

2. Do you use Bloom's Taxonomy as part of your planning for tests? 

3 . Do you write contrait outlines as part of your planning for tests? 

4. Do you write test-item specifications as part of your planning for tests? 

5. Which short-answer test-itm types do you use in testing?: 

a. unstructured (can be answered by a word, phrase, or number) 

b. completion (fill in an omitted word or phrase) 

c. true-frlse (yes-no) 

d. two-choice clasdfication 

e. multiple choice 
f matching 

6. Do you use essay-type test itons in testing? 

7. Do you use performance-type tests? 

If so, what kinds (eg., writing, dramatic presentations, science projects, portfolios)? 



O 



8. What means, if any, do you use to insure that your tests have content validity? 



9. What means, if any, do you use to build reliability into your tests? 

10. Do you use standardized tests? 

1 1 . What types of test items (eg., unstructured, completion, true-false, two-choice, multiple choice, matching, 
essay) do you most frequently use on your teachra^-made tests? 

1 2. a. How are norm-referenced tests, like the Stanford, useful? 

b. What are their drawbacks? 

13. a. How are criterion-referenced tests, like the MPT, useful? 
b. What are ther drawbadcs? 

14. What types of standard scores do you use or have need to be able to intepret? 

a. z-scores 

b. T-scores 

c. CEEB scores 

d. AGCT scores 

15. Do you need to be able to interpret stanine scores? 

16. Do you need to be able to interpret percentile ranks? 

17. Do you need to be able to interpret grade-equival«it scores? 

18. Do you use or need to be able to interpret standard deviations? 

19. Do you need to be able to int^ret Wechsler Scales? 

20. Do you have any recommendations concerning the t^ching of the course in Diagnostic and Evaluative 
Procures in terms of any content or otho* aspects? 

Thank you very much for your help. Your comments will contrilmte to the quality of the course. We appreciate 
your time and droughts. (Be sure to be enthusiastic in expr^ing your appreciation. They did you a fovor.) 

O 
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Response Posttest 

Expected Mean Squares Section 



Source 




Term 


Denominator Expected 


Term 




DF Fixed? 


Term 


Mean Square 


A(GroupE=1,T=2) 




1 Yes 


S(A) 


S+sA 


S(A) 




140 No 




S 


Note: Expected Mean Squares are for the balanced cell-frequency case. 


Analysis of Variance Table 








Source 




Sum of 


Mean 


Prob 


Term 


DF 


Squares 


Square 


F-Ratio Level 


X(Pretest) 


1 


359.6284 


359.6284 


44.60 0.000000* 


A(GroupE=1,T=2) 


1 


11.03386 


11.03386 


1.37 0244100 


S 


140 


1128.99 


8.064211 




Total (Adjusted) 


142 


1489J664 






Total 


143 








* Term significant at alpha = 0.05 






Means and Standard Error Section 














Standard 


Term 




Count 


Mean 


Error 


All 




143 


1326816 




A:GroupE=1,T=2 










1 




78 


13.54918 


0.321539 


2 




65 


12.98714 


0.3522283 



Plots Section 





6 



Power 

(Alpha=0.05) 

0.999998 

0212094 
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